Imaging apparatus and imaging method with error corrected interpolation frame generation

ABSTRACT

An imaging apparatus includes an imaging unit and a transmission unit. The imaging unit is configured to capture two images that are different from each other by a predetermined amount of an optical distance (focus) between an objective lens and an imaging device having a first resolution. The transmission unit is configured to transmit the captured images.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a Continuation of application Ser. No. 15/052,705, filed Feb.24, 2016, which is a Continuation of application Ser. No. 14/252,039,filed on Apr. 14, 2014, now U.S. Pat. No. 9,294,663, issued Mar. 22,2016, which contains subject matter related to Japanese Priority PatentApplication JP 2013-098446 filed May 8, 2013, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an imaging apparatus and an imagingmethod that achieve a high resolution image, a hyperspectral image, astereoscopic image, and a refocused image based on two images withdifferent focuses.

In order to achieve high resolution of images in related art, there isan enlargement technique in which upsampling is performed on theoriginal image for enlargement and jaggies generated on the image due tothe enlargement are smoothed using an appropriate filter.

Additionally, there is a super-resolution technique of interpolatinglacking information between frames of a video, interpolatingpseudo-pixels using a heuristics database, or showing an image in anergonomically beautiful manner by increasing image information using anintelligent method (see, for example, Japanese Patent ApplicationLaid-open No. 2009-181508).

Further, a technique of pixel shift is exemplified as a high resolutiontechnique in which an imaging device with a certain resolution is usedto capture an image with a resolution exceeding the resolution of theimaging device (see, for example, Japanese Patent Application Laid-openNo. 2013-12112).

Further, a hyperspectral camera can capture an image including spectraof a lot of frequency bands in the range from visible light tonear-infrared light (see, for example, Japanese Patent ApplicationLaid-open No. 2001-521772)

Furthermore, a camera by an integral imaging method that is capable ofvertically and horizontally moving a perspective is created as asubsequent step of a stereo 3D camera (see, for example, Japanese PatentApplication Laid-open No. 2008-219788)

Moreover, a light field camera capable of generating an image whosefocal position is freely changed (refocused) after the image is capturedis created (see, for example, Japanese Patent Application Laid-open No.2009-224982). Hereinafter, the image whose focal position is freelychanged is referred to as a “refocused image”.

SUMMARY

A camera has a basic function of three-color monochrome photographybased on three primary colors of light. For example, in a frequency bandof red light, the camera has difficulty of determining a difference inabout 10 nm between different spectra. This is because, when the sum ofluminance values of an R channel falls in a single luminance level, theluminance values are rounded in the same R channel.

Due to this rounding, it has been difficult to use a normal camera foran image analysis in a case where an analysis of a slight colordifference is necessary, such as in medical use.

Additionally, as cameras to observe such information, a hyperspectralcamera and a multispectral camera are in practical use. However, thosecameras are hard to use because of performing only one-dimensionalimaging at a time and performing imaging exclusively in a specificspectrum.

In the integral imaging method, lens arrays are used and thus there areproblems that the resolution of captured images is poor and the dataamount of the captured images becomes huge.

As described above, in the methods of related art, it has been difficultto easily acquire a high resolution image, a hyperspectral image, astereoscopic image, and a refocused image.

In view of the circumstances as described above, it is desirable toprovide an imaging apparatus and an imaging method that are capable ofeasily acquiring a high resolution image, a hyperspectral image, astereoscopic image, and a refocused image.

According to an embodiment of the present disclosure, there is providedan imaging apparatus including an imaging unit and a transmission unit.The imaging unit is configured to capture two images that are differentfrom each other by a predetermined amount of an optical distance (focus)between an objective lens and an imaging device having a firstresolution. The transmission unit is configured to transmit the capturedimages.

In the embodiment of the present disclosure, in the imaging apparatus,two images that are different from each other by a predetermined amountof focus and have a first resolution are acquired and transmitted to thedevelopment apparatus.

In a development apparatus for a high resolution development, the anglesof view of the two images are equalized, one of the two images being afirst focus image and the other image being a second focus image. Animage obtained by diffusing and enlarging the first focus image to havea second resolution higher than the first resolution is generated as adigital defocused image. Further, an image obtained by upsampling thesecond focus image at the second resolution is generated as an enlargeddefocused image. Furthermore, a difference for each pixel between thedigital defocused image and the enlarged defocused image is generated asan interference image. By a learning-type pattern conversion circuit, ahigh-frequency component of the second resolution is generated from theinterference image, and component synthesis processing is performed withthe first focus image being regarded as a low-frequency component, todevelop an image having the second resolution.

In a development apparatus for a hyperspectral development, the anglesof view of the two images are equalized, one of the two images being afirst focus image and the other image being a second focus image. Animage obtained by diffusing the first focus image is generated as adigital defocused image. A difference for each pixel between the digitaldefocused image and the second focus image is generated as aninterference image. Through processing of emphasizing a magnificationchromatic aberration by a learning-type pattern conversion circuit, ahigh-frequency component containing spectral information is generatedfrom the interference image, and component synthesis processing of thefirst focus image and the high-frequency component is performed, todevelop a hyperspectral image.

In a development apparatus for an image for stereoscopic viewing, theangles of view of the two images are equalized, one of the two imagesbeing a first focus image and the other image being a second focusimage. An image obtained by diffusing the first focus image by a firstfunction is generated as a first digital defocused image, and an imageobtained by diffusing the first focus image by a second function isgenerated as a second digital defocused image, the first function andthe second function being linearly symmetrical with each other. Adifference for each pixel between the first digital defocused image andthe second focus image is generated as first mask information, and adifference for each pixel between the second digital defocused image andthe second focus image is generated as second mask information. A firstimage for stereoscopic viewing is developed from the first focus imageand the first mask information, and a second image for stereoscopicviewing is developed from the first focus image and the second maskinformation.

In a development apparatus for a refocused image development, the firstfocus image and the second focus image are taken in and a difference foreach pixel between the two images is generated as an interference image.By a learning-type pattern conversion circuit, a high-frequencycomponent is generated from luminance information and spectralinformation of the interference image, and component synthesisprocessing of the first focus image and the high-frequency component isperformed, to develop a refocused image.

Through the processing described above, in the present disclosure, it ispossible to easily acquire a high resolution image, a hyperspectralimage, a stereoscopic image, and a refocused image.

According to the embodiment of the present disclosure, in the imagingapparatus, the imaging unit may include a formed-image-capturing deviceconfigured to capture a formed image on which light passing through theobjective lens is focused, and a defocused-image-capturing deviceconfigured to capture a defocused image on which the light is defocusedbased on the predetermined amount.

According to the embodiment of the present disclosure, the imagingapparatus may further include: an angle-of-view adjustment unitconfigured to equalize angles of view of the two images, one of the twoimages being a first focus image and the other image being a secondfocus image; a diffusion unit configured to generate, as a digitaldefocused image, an image obtained by diffusing and enlarging the firstfocus image to have a second resolution higher than the firstresolution; an upsampling unit configured to generate, as an enlargeddefocused image, an image obtained by upsampling the second focus imageto have the second resolution; and a difference calculation unitconfigured to generate, as an interference image, a difference for eachpixel between the digital defocused image and the enlarged defocusedimage, in which the transmission unit may be configured to transmit thefirst focus image and the interference image.

According to the embodiment of the present disclosure, in the imagingapparatus, the first focus image may be a formed image that is in focus,and the second focus image may be a defocused image that is defocused bya predetermined amount from the in-focus position.

According to the embodiment of the present disclosure, in the imagingapparatus, the diffusion unit may be configured to diffuse the formedimage by a point spread function.

According to the embodiment of the present disclosure, the imagingapparatus may further include: an angle-of-view adjustment unitconfigured to equalize angles of view of the two images, one of the twoimages being a first focus image and the other image being a secondfocus image; a diffusion unit configured to generate, as a digitaldefocused image, an image obtained by diffusing the first focus image;and a difference calculation unit configured to generate, as aninterference image, a difference for each pixel between the digitaldefocused image and the second focus image, in which the transmissionunit may be configured to transmit the first focus image and theinterference image.

According to the embodiment of the present disclosure, the imagingapparatus may further include: an angle-of-view adjustment unitconfigured to equalize angles of view of the two images, one of the twoimages being a first focus image and the other image being a secondfocus image; a diffusion unit configured to generate, as a first digitaldefocused image, an image obtained by diffusing the first focus image bya first function and generate, as a second digital defocused image, animage obtained by diffusing the first focus image by a second function,the first function and the second function being linearly symmetricalwith each other; and a difference calculation unit configured togenerate, as first mask information, a difference for each pixel betweenthe first digital defocused image and the second focus image andgenerate, as second mask information, a difference for each pixelbetween the second digital defocused image and the second focus image,in which the transmission unit may be configured to transmit the firstfocus image, the first mask information, and the second maskinformation.

According to the embodiment of the present disclosure, the imagingapparatus may further include: an angle-of-view adjustment unitconfigured to equalize angles of view of the two images, one of the twoimages being a first focus image and the other image being a secondfocus image; and a difference calculation unit configured to generate,as an interference image, a difference for each pixel between the firstfocus image and the second focus image, in which the transmission unitmay be configured to transmit the first focus image, the second focusimage, and the interference image.

According to an embodiment of the present disclosure, there is providedan imaging apparatus including: an imaging unit configured to capture animage; a correction unit configured to perform at least one of anaberration correction and a digital optical correction on the image togenerate a corrected image; a difference calculation unit configured togenerate, as an interference image, a difference for each pixel betweenthe image and the corrected image; and a transmission unit configured totransmit coordinate information contained in the image, the interferenceimage, and the corrected image.

According to an embodiment of the present disclosure, there is providedan imaging method including: capturing two images that are differentfrom each other by a predetermined amount of an optical distance (focus)between an objective lens and an imaging device; and transmitting thecaptured images.

As described above, according to the present disclosure, it is possibleto easily acquire a high resolution image, a hyperspectral image, astereoscopic image, and a refocused image.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a configuration diagram showing an overall configuration of animaging apparatus and a development apparatus according to an embodimentof the present disclosure;

FIG. 2 is a configuration diagram showing an overall configuration ofanother imaging apparatus and another development apparatus according tothe embodiment of the present disclosure;

FIG. 3 is a diagram for describing a concept of a high resolutiondevelopment;

FIG. 4 is a diagram showing what effects are provided by the highresolution development according to the embodiment of the presentdisclosure;

FIG. 5 is a diagram for describing a concept of generating a highresolution image by an inverse operation using a defocused image;

FIG. 6 is a diagram showing a defocus amount;

FIG. 7 is a diagram for describing a specific example of the defocusmethod;

FIG. 8 is a diagram showing component separation processing andcomponent synthesis processing;

FIG. 9 is a diagram showing implementation examples of the imagingapparatus;

FIG. 10 is a diagram showing that a 4K image as a reference is used togenerate a first focus image and a second focus image, and the highresolution development is performed on the first focus image and thesecond focus image to generate a developed image through an interferenceimage and a high-frequency component;

FIG. 11 is a diagram showing images obtained by enlarging black-eyeparts of the images shown in FIG. 10;

FIG. 12 is a diagram showing that an 8K image as a reference is used togenerate a first focus image and a second focus image, and the highresolution development is performed on the first focus image and thesecond focus image to generate a developed image through an interferenceimage and a high-frequency component;

FIG. 13 is a diagram showing a state where information on Δ amount isseparated from the interference image by an algorithm to develop a highresolution image by ΔY, a hyperspectral image by Δω, and a stereoscopicimage by Δφ;

FIG. 14 is a configuration diagram showing an overall configuration ofthe imaging apparatus and another development apparatus according toanother embodiment of the present disclosure;

FIG. 15 is a configuration diagram showing an overall configuration ofanother imaging apparatus and another development apparatus according tothe embodiment of the present disclosure;

FIG. 16 is a diagram for describing a hyperspectral development;

FIG. 17 is a diagram showing a specific example of the hyperspectraldevelopment;

FIG. 18 is a configuration diagram showing an overall configuration ofthe imaging apparatus and another development apparatus according toanother embodiment of the present disclosure;

FIG. 19 is a configuration diagram showing an overall configuration ofanother imaging apparatus and another development apparatus according tothe embodiment of the present disclosure;

FIG. 20 is a diagram showing a processing flow of a stereoscopicdevelopment;

FIG. 21 is a diagram for describing the stereoscopic development;

FIG. 22 is a configuration diagram showing an overall configuration ofthe imaging apparatus and another development apparatus according toanother embodiment of the present disclosure;

FIG. 23 is a configuration diagram showing an overall configuration ofanother imaging apparatus and another development apparatus according tothe embodiment of the present disclosure;

FIG. 24 is a diagram for describing the principles of a refocusingdevelopment;

FIG. 25 is a diagram showing a state where an optical path is notestimated by using only luminance information;

FIG. 26 is a diagram showing a state where the optical path can beestimated by using the luminance information and spectral information;

FIG. 27 is a diagram showing a state where a condition of amagnification chromatic aberration differs due to an in-focus positionmoving backward and forward;

FIG. 28 is a diagram showing examples of the spectral information in theinterference image and showing how the point spread function is causedto act in a diffusing or converging direction for each example;

FIG. 29 is a diagram showing an example of an image in which not a dollbut a background is in focus;

FIG. 30 is a diagram showing an example of an image in which not thedoll but a front side is in focus;

FIG. 31 is a diagram showing an example of a refocused image obtained byperforming the refocusing development on the images shown in FIGS. 29and 30 to develop an image with the doll being in focus;

FIG. 32 is a diagram showing an example of a deep-focus image obtainedby performing the refocusing development on the images shown in FIGS. 29and 30 to develop an image with the doll and the background being infocus;

FIG. 33 is a configuration diagram showing an overall configuration ofanother imaging apparatus and another development apparatus according toanother embodiment of the present disclosure;

FIG. 34 is a diagram showing that a 4K image as a reference is used togenerate a first focus image, and the high resolution development isperformed on the first focus image to generate a developed image;

FIG. 35 is a diagram showing a relationship between a normal shutterspeed of 1/60 seconds, an image captured at 1/N of the normal shutterspeed, and an image captured at a shutter speed of (N−1)/60 N secondsobtained by subtracting 1/60N seconds from 1/60 seconds;

FIG. 36 is a diagram for describing the principles of a high frame ratedevelopment; and

FIG. 37 is a functional block diagram showing a configuration of anotherimaging apparatus.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be describedwith reference to the drawings.

First Embodiment

First, the overall configuration will be described. Subsequently, thegeneral outline of a technique of achieving high resolution of an imageaccording to an embodiment of the present disclosure (hereinafter,referred to as high resolution development) will be described. Finally,specific examples will be described.

It should be noted that the high resolution development according to theembodiment of the present disclosure is for collecting interferenceinformation of light from two images whose focal points are shifted onthe same optical axis and developing an image with a resolution higherthan a resolution of an imaging device based on the interferenceinformation.

[Overall Configuration]

First, the overall configuration of an imaging apparatus and adevelopment apparatus according to the embodiment of the presentdisclosure will be described. FIG. 1 is a configuration diagram showingan overall configuration of an imaging apparatus 100 and a developmentapparatus 200 according to the embodiment of the present disclosure. Itshould be noted that the terms used below, i.e., a “first focus” and a“second focus”, typically refer to a focus in an in-focus state and afocus defocused from the focus in the in-focus state by a predeterminedamount. In this case, a “first focus image” can be read as a “formedimage” and a “second focus image” can be read as a “defocused image”.

However, the focuses represented by the “first focus” and the “secondfocus” are not limited to the above. A focus that is defocused from thefocus in the in-focus state to a subject side by a predetermined amountmay be referred to as a “first focus”, and a focus that is defocusedfrom the focus in the in-focus state to an image plane side by apredetermined amount may be referred to as a “second focus”.

The imaging apparatus 100 includes an objective lens 1, a half mirror 2,an imaging device (first focus imaging device, formed-image-capturingdevice) 3, a first-focus-image-capturing controller (imaging unit) 4, anoptical path length change filter 5, an imaging device (second focusimaging device, defocused-image-capturing device) 6, asecond-focus-image-capturing controller (imaging unit) 7, and atransmission unit 8.

The objective lens 1 is an optical system for forming an image of asubject on the imaging device in order to capture the image of thesubject, as in the case of a general digital camera.

The half mirror 2 splits light that has passed through the objectivelens 1 in order that the imaging device 3 and the imaging device 6receive the light.

The imaging device 3 captures an image formed on an imaging plane in thefirst focus of the objective lens 1.

The first-focus-image-capturing controller 4 acquires the image capturedby the imaging device 3. It should be noted that the image acquired bythe imaging device 3 and the first-focus-image-capturing controller 4are hereinafter referred to as a first focus image. In the first focusimage, a high-frequency component of the subject, which is not capturedby one pixel of the imaging device 3, is rounded in one pixel.

The optical path length change filter 5 is a filter for preciselyadjusting an optical path length from the objective lens 1 to theimaging device 6. It should be noted that in this embodiment, theoptical path length change filter 5 is used to adjust the optical pathlength, but in place of this, for example, a configuration to adjust theoptical path length by adjusting the position of the imaging device 6may be provided.

The imaging device 6 captures an image defocused (blurred) by theoptical path length change filter 5 in the second focus.

The second-focus-image-capturing controller 7 acquires the imagecaptured by the imaging device 6. It should be noted that the imageacquired by the imaging device 6 and the second-focus-image-capturingcontroller 7 is hereinafter referred to as a second focus image. In thesecond focus image, a high-frequency component of the subject, which isnot captured by one pixel of the imaging device 6, is diffused over aplurality of pixels and comes out in a blurred form.

It should be noted that the first-focus-image-capturing controller 4 andthe second-focus-image-capturing controller 7 may be integrated as oneimaging controller.

The transmission unit 8 transmits the first focus image supplied fromthe first-focus-image-capturing controller 4 and the second focus imagesupplied from the second-focus-image-capturing controller 7 to thedevelopment apparatus 200.

In the above configuration, the half mirror 2 and the two imagingdevices 3 and 6 are used to acquire the first focus image and the secondfocus image at the same time. In the case where the two images do nothave to be acquired at the same time, however, a mechanism to add theoptical path length change filter 5 to the optical path or remove theoptical path length change filter 5 from the optical path may beprovided, so that the half mirror 2 may be removed and the first focusimage and the second focus image may be acquired by one imaging device.

Subsequently, the configuration of the development apparatus 200 will bedescribed.

The development apparatus 200 includes a reception unit 9, anangle-of-view adjustment unit 10, a diffusion processing unit (diffusionunit) 11, an upsampling processing unit (upsampling unit) 12, adifference calculation unit 13, a de-blurring processing unit 14, and acomponent synthesis processing unit 15.

The reception unit 9 receives the first focus image and the second focusimage, which are transmitted from the transmission unit 8 of the imagingapparatus 100. Both of the received images are supplied to theangle-of-view adjustment unit 10.

The angle-of-view adjustment unit 10 adjusts the angle of view of eachof the first focus image and the second focus image that have slightlydifferent angles of view because of the different focuses, toappropriately perform the following processing for each pixel. Theadjustment of the angle of view may be performed on only one of thefirst focus image and the second focus image.

The diffusion processing unit 11 uses a point spread function to changethe first focus image into an image in a defocused state by computing.This image is hereinafter referred to as a digital defocused image.Through this processing, the first focus image is changed into adigitally blurred image. In the case where the original first focusimage is an image with an HD (High Definition) size and if an image tobe developed is a 4K resolution image, for example, the digitaldefocused image is enlarged to be an image with a 4K size. In the casewhere the image to be developed is an image with an 8K size, thediffusion processing unit 11 performs defocusing to acquire an 8K sizeimage.

The upsampling processing unit 12 performs upsampling on the secondfocus image. For example, in the case where the second focus image is anHD size image and an image to be developed is a 4K resolution image, theupsampling processing unit 12 doubles the number of pixels verticallyand horizontally arranged to obtain a 4K size image. The image obtainedby the upsampling is hereinafter referred to as an enlarged defocusedimage. In the case where the image to be developed is an 8K size image,the upsampling processing unit 12 performs upsampling to obtain an 8Ksize image.

The difference calculation unit 13 calculates a difference value foreach pixel between the digital defocused image, which is generated bythe diffusion processing unit 11, and the enlarged defocused image,which is generated by the upsampling processing unit 12. The calculationresult is hereinafter referred to as an interference image.

The de-blurring processing unit 14 is constituted of a learning-typepattern conversion circuit and performs de-blurring processing on theinterference image supplied from the difference calculation unit 13 togenerate a high-frequency component. The de-blurring processing unit 14regards this interference image as a defocused image and generates ahigh-frequency component of an image with a resolution higher than theresolution of the first focus image by an inverse operation method forthe original image before the defocusing.

The component synthesis processing unit 15 synthesizes thehigh-frequency component, which is generated by the de-blurringprocessing unit 14, and a low-frequency component (first focus image),which is supplied from the reception unit 9, into one image by componentsynthesis processing. The synthesized image is hereinafter referred toas a developed image.

Hereinabove, the overall configuration has been described.

[Modified Example of Overall Configuration]

Next, an imaging apparatus 101 and a development apparatus 201 that haveconfigurations different from the above configurations will bedescribed. FIG. 2 is a configuration diagram showing an overallconfiguration of the imaging apparatus 101 and the development apparatus201 according to the embodiment of the present disclosure.

The main difference from the configurations described above is that theangle-of-view adjustment unit 10, the diffusion processing unit 11, theupsampling processing unit 12, and the difference calculation unit 13 ofthe development apparatus 200 are moved to the imaging apparatus 101side. It should be noted that constituent elements having the samefunctions as those of the above constituent elements are denoted by thesame reference symbols and description thereof will be omitted.

The imaging apparatus 101 includes an objective lens 1, a half mirror 2,an imaging device 3, a first-focus-image-capturing controller 4, anoptical path length change filter 5, an imaging device 6, asecond-focus-image-capturing controller 7, the angle-of-view adjustmentunit 10, the diffusion processing unit 11, the upsampling processingunit 12, the difference calculation unit 13, and a transmission unit 8B.

The transmission unit 8B transmits an interference image, which isgenerated by the difference calculation unit 13, and a low-frequencycomponent (first focus image) supplied from thefirst-focus-image-capturing controller 4, to the development apparatus201.

The development apparatus 201 includes a reception unit 9B, ade-blurring processing unit 14, and a component synthesis processingunit 15.

The reception unit 9B receives the interference image and thelow-frequency component transmitted from the transmission unit 8B of theimaging apparatus 101. The received interference image is supplied tothe de-blurring processing unit 14, and the received low-frequencycomponent is supplied to the component synthesis processing unit 15.

Hereinabove, the modified example of the overall configuration has beendescribed.

[Concept of High Resolution Development]

Subsequently, a concept of the high resolution development will bedescribed. FIG. 3 is a diagram for describing the concept of the highresolution development.

One pixel of the imaging device is regarded as a black box, and light isapplied to the black box from the side with a flashlight to generate ashadow outside the one pixel. The shadow is to be a signal that is finerthan the one pixel and leaks out to the next pixel.

The signal that is finer than the one pixel is not observed normally.When the light is applied from the side with the flashlight, however,the shadow of that finer signal appears on an adjacent pixel. The highresolution development according to the embodiment of the presentdisclosure uses the fact that, like this shadow, a trace of the signalfiner than the one pixel leaks out to the adjacent pixel when the focusis shifted, to develop information finer than the one pixel based on thetrace.

Subsequently, the high resolution development according to theembodiment of the present disclosure will be described from a differentperspective. FIG. 4 is a diagram showing what effects are provided bythe high resolution development according to the embodiment of thepresent disclosure.

FIG. 4 shows, on its left part, the size of subjects, that is, a subjectA corresponding to one pixel and a subject B corresponding to fourpixels of the 4K resolution. FIG. 4 shows, on its center part, the sizeof the imaged subjects A and B. When those subjects A and B are imagedwith an HD camera, both of an image A and an image B are expressed inthe size corresponding to one pixel of the HD resolution. FIG. 4 shows,on its right part, that the development in the 4K size is performed bythe high resolution development based on the trace (shadow) of the lightand thus the image of the subject A is restored to be the imagecorresponding to one pixel of the 4K resolution.

Subsequently, the concept of generating a high resolution image by aninverse operation using a defocused image will be described. FIG. 5 is adiagram for describing the concept of generating the high resolutionimage by the inverse operation using the defocused image.

With reference to the left part of FIG. 5, when an image of the subjectis formed on an image-forming plane A, the image occupies an areacorresponding to one pixel, but on a blur plane B that is defocused byshifting the image-forming plane A, a blurred image is formed over thenext pixels and captured in the size corresponding to three pixels. Thismeans that the high-frequency component, which is too finer than the onepixel of the imaging device to be observed in the one pixel, is observedas blurred information. FIG. 5 shows, on its right part, that an imagewith a resolution higher than the original resolution of the imagingdevice is developed on a virtual image-forming plane C by an inverseoperation using the interference image.

Hereinabove, the concept of the high resolution development has beendescribed.

[System of High Resolution Development]

In general, it is difficult for the imaging device to image informationsmaller than an imaging pixel, and if the subject has complexinformation, the information smaller than the imaging pixel is roundedas information on one pixel of the imaging device. Meanwhile, when thefocal point is moved, the image of the subject is diffused to extendover adjacent imaging pixels, and thus the information that is notcaptured by the one pixel of the imaging pixel comes out also on theadjacent imaging pixels.

In the embodiment of the present disclosure, a difference between thedigital defocused image and an actual defocused image is obtained. Thedigital defocused image is obtained by diffusing the formed image by thepoint spread function so as to obtain a predetermined blur amount. As aresult, the information on the high-frequency component that is notcaptured by the one imaging pixel remains in the interference image,which is obtained from the difference, in the form where the image isdiffused. This information is subjected to an inverse operation for thevirtual image-forming plane, so that the finer high-frequency componentthat is not captured by the one imaging pixel is calculated and theimage information finer than the resolution of the imaging device isdeveloped.

[Inverse Operation Method for Virtual Image-Forming Plane]

The obtained interference image and the actual high-frequency componenthave a geometric correlativity, and an input signal and an instructionalsignal desired to be eventually acquired are given to the learning-typepattern conversion circuit to perform reinforcement learning, thusestablishing an inverse operation module (de-blurring processing unit14). With this inverse operation module, it has been found that an imagewith a resolution higher than that of the imaging device can beobtained.

[Defocus Amount]

Subsequently, the defocus amount when the defocused image is capturedwill be described. FIG. 6 is a diagram showing the defocus amount.

Each of the rectangles shown in FIG. 6 expresses a pixel of the imagingdevice. The innermost circle is a circle of least confusion of theoptical system in the imaging apparatus 100 or 101.

For example, in the case where the imaging device has the HD resolutionand an image to be developed is a 4K resolution image, defocusing isperformed until the size of the circle of confusion is changed into thesize of a circle indicated by “2K-6K” in FIG. 6.

Further, for example, in the case where the imaging device has the HDresolution and an image to be developed is an 8K resolution image,defocusing is performed until the size of the circle of confusion ischanged into the size of a circle indicated by “6K-10K” in FIG. 6.

In the above example, the resolutions of HD, 4K, and 8K are exemplified.In the case of other resolutions, defocusing may be performed at thesame ratio as in the case of the resolutions of HD, 4K, and 8K.

Hereinabove, the defocus amount has been described.

[Defocus Method]

Subsequently, a specific example of the defocus method will bedescribed. FIG. 7 is a diagram for describing a specific example of thedefocus method.

Although the optical path length change filter 5 is already described inthe section of the overall configuration, as described regarding thedefocus amount, the defocus amount has to be adjusted depending on theresolution of an image to be developed. A filter disk is exemplified asa specific example of an adjustment of the defocus amount.

As shown in FIG. 7, for example, optical path length change filters 5for developing images with resolutions of 3.2K, 4K, and 8K are fittedinto a filter disk. The imaging apparatus 100 or 101 rotates the filterdisk in accordance with the resolution of an image to be developed, anduses an appropriate optical path length change filter 5 to perform ahigh resolution development corresponding to a target resolution.

Hereinabove, the specific example of the defocus method has beendescribed.

[Component Synthesis Processing]

Subsequently, the component synthesis processing will be describedtogether with component separation processing. FIG. 8 is a diagramshowing the component separation processing and the component synthesisprocessing.

First, the component separation processing will be described. It isassumed that a high resolution image A includes four pixels D1, D2, D3,and D4. In the component separation processing, a low-frequencycomponent C and a high-frequency component B are generated by thefollowing expressions, where the high-frequency component B isconstituted of pixel values D1′, D2′, D3′, and D4′:Low-frequency component C=Pixel mean value Dm=(D1+D2+D3+D4)/4;D1′=D1−Dm;D2′=D2−Dm;D3′=D3−Dm; andD4′=D4−Dm.

The component synthesis processing is an inverse operation of thecomponent separation processing. Pixel values D1 to D4 of a highresolution image D to be obtained can be obtained by the followingexpressions:D1=D1′+Dm;D2=D2′+Dm;D3=D3′+Dm; andD4=D4′+Dm.Here, the component synthesis processing is performed assuming the firstfocus image (of HD size) to be a low-frequency component and aprocessing result of the de-blurring processing unit 14 to be ahigh-frequency component, and thus a high resolution image (of 4K size)can be obtained as a developed image.

Hereinabove, the component synthesis processing has been described.

[Implementation Examples of Imaging Apparatus]

Next, implementation examples of the imaging apparatus will bedescribed. FIG. 9 is a diagram showing the implementation examples ofthe imaging apparatus.

A “built-in-type large-dual-sensor camera” on the upper left part ofFIG. 9 has a configuration that is the most similar to the overallconfiguration shown in FIG. 1 and FIG. 2.

A “half-mirror-rig-system stereo camera” on the upper right part of FIG.9 has a configuration in which objective lenses 1 and cameras 300 arearranged on the rear side of a half mirror 2 within a beam splitter 301and are fixed by a rig 302.

A “large-lens-spectral-type stereo camera” on the lower right part ofFIG. 9 has a configuration in which a half mirror 2 is arranged inside alarge lens 303.

A “parallel-system stereo camera” on the lower left part of FIG. 9 isfor defocusing one of images captured with left and right objectivelenses 1. The point of sight shifts between the left and right imagesand this limits the enhancement of resolution, but this camera producescertain effects.

It should be noted that those implementation examples are also appliedto the embodiments described below, in addition to the first embodiment.

Hereinabove, the implementation examples of the imaging apparatus havebeen described.

Specific Examples

Next, an example in which the high resolution development is performedusing the high resolution development technique according to theembodiment of the present disclosure will be described.

FIG. 10 is a diagram showing that a 4K image A as a reference is used togenerate a first focus image B and a second focus image C, and the highresolution development is performed on the first focus image B and thesecond focus image C to generate a developed image F through aninterference image D and a high-frequency component E. Comparing thereference 4K image A with the developed image F, it is found that thereis not a large difference therebetween. However, in the high resolutiondevelopment performed herein, a specific pattern is intensively learnedin the learning-type pattern conversion circuit.

FIG. 11 is a diagram showing images obtained by enlarging black-eyeparts of the images shown in FIG. 10. Comparing a reference image A witha developed image C, it is found that the development accuracy of thedeveloped image C is about nine-tenths of the development accuracy ofthe reference image A.

FIG. 12 is a diagram showing that an 8K image A as a reference is usedto generate a first focus image B and a second focus image C, and thehigh resolution development is performed on the first focus image B andthe second focus image C to generate a developed image F through aninterference image D and a high-frequency component E. Comparing thereference 8K image A with the developed image F, it is found that thereis not a large difference therebetween. However, in the high resolutiondevelopment performed herein, a specific pattern is intensively learnedin the learning-type pattern conversion circuit.

Hereinabove, the first embodiment has been described.

Second Embodiment

The first embodiment achieves the high resolution of an image byacquiring the high-frequency component from the interference image. Now,the information acquired from the interference image is summed up.

In general, undulation is expressed by a mathematical expression ofY*sin(ωt+φ), and when the undulation is regarded as image information, Yis interpreted as luminance, ω is interpreted as spectral information ofRGB, and φ is interpreted as stereoscopic information.

With a general camera in related art, only luminance informationresolved into information on ergonomic three primary colors of RGB oflight is developed.

The interference image obtained from the two images, i.e., the firstfocus image (formed image) and the second focus image (defocused image),contains information on change amounts of Y, ω, and φ of the first focusimage serving as a reference image. The inventor of the presentdisclosure found that the information on those change amounts can beseparated from the interference image by an algorithm to develop a highresolution image by ΔY, a hyperspectral image by Δω, and a stereoscopicimage or a refocused image by Δφ (see FIG. 13).

The high resolution development technique described in the firstembodiment achieves a resolution exceeding the resolution of the imagingdevice by adding the information of ΔY to the reference image.

Description will be given on a hyperspectral development using Δω in thesecond embodiment, on a stereoscopic development using Δφ in a thirdembodiment, and on a refocusing development using Δφ in a fourthembodiment. Here, those development techniques are collectively referredto as a holographic development.

[Overall Configuration]

First, the overall configuration of an imaging apparatus and adevelopment apparatus according to an embodiment of the presentdisclosure will be described. FIG. 14 is a configuration diagram showingthe overall configuration of an imaging apparatus 100 and a developmentapparatus 202 according to the embodiment of the present disclosure.

The configuration of the imaging apparatus 100 to be used in the secondembodiment is the same as that of the first embodiment, and thusdescription thereof will be omitted.

The main difference on the development apparatus from the firstembodiment is that the development apparatus 200 is replaced with thedevelopment apparatus 202, in which the upsampling processing unit 12 isremoved and the diffusion processing unit 11, the difference calculationunit 13, and the de-blurring processing unit 14 are provided withdifferent functions to be changed into a diffusion processing unit 11B,a difference calculation unit 13B, and a de-blurring processing unit14B, respectively. In the following description, constituent elementshaving the same functions as those of the first embodiment are denotedby the same reference symbols and description thereof will be omitted.

The development apparatus 202 includes a reception unit 9, anangle-of-view adjustment unit 10, the diffusion processing unit 11B, thedifference calculation unit 13B, the de-blurring processing unit 14B,and a component synthesis processing unit 15.

The diffusion processing unit 11B uses a point spread function to changethe first focus image into a digital defocused image in a statedefocused by computing. Through this processing, the first focus imageis changed into a digitally blurred image. However, unlike the diffusionprocessing unit 11 of the first embodiment, the diffusion processingunit 11B does not enlarge the original first focus image, e.g., changethe HD resolution to the 4K or 8K resolution.

The difference calculation unit 13B calculates a difference value foreach pixel between the digital defocused image, which is generated bythe diffusion processing unit 11B, and the second focus image, which issupplied from the angle-of-view adjustment unit 10, to generate aninterference image.

The de-blurring processing unit 14B is constituted of a learning-typepattern conversion circuit and performs de-blurring processing on aninterference image supplied from the difference calculation unit 13B togenerate a high-frequency component. The de-blurring processing unit 14regards this interference image as a defocused image and generates ahigh-frequency component of an image with a resolution higher than theresolution of the formed image by an inverse operation method for theoriginal image before the defocusing. In the de-blurring processing unit14B, however, learning by the learning-type pattern conversion circuitis performed so as to further emphasize contrast, compared with thede-blurring processing performed by the de-blurring processing unit 14of the first embodiment.

Hereinabove, the overall configuration has been described.

[Modified Example of Overall Configuration]

Next, an imaging apparatus 102 and a development apparatus 203 that haveconfigurations different from the above configurations will bedescribed. FIG. 15 is a configuration diagram showing the overallconfiguration of the imaging apparatus 102 and the development apparatus203 according to the embodiment of the present disclosure.

The main difference from the configurations described above is that theangle-of-view adjustment unit 10, the diffusion processing unit 11B, andthe difference calculation unit 13B of the development apparatus 202 aremoved to the imaging apparatus 102 side. It should be noted thatconstituent elements having the same functions as those of the aboveconstituent elements are denoted by the same reference symbols anddescription thereof will be omitted.

The imaging apparatus 102 includes an objective lens 1, a half mirror 2,an imaging device 3, a first-focus-image-capturing controller 4, anoptical path length change filter 5, an imaging device 6, asecond-focus-image-capturing controller 7, the angle-of-view adjustmentunit 10, the diffusion processing unit 11B, the difference calculationunit 13B, and a transmission unit 8B.

The development apparatus 203 includes a reception unit 9B, ade-blurring processing unit 14B, and a component synthesis processingunit 15.

Hereinabove, the modified example of the overall configuration has beendescribed.

[Hyperspectral Development]

A color camera in related art resolves incident light into channels ofthe three primary colors of RGB of light and captures a full-color imageby performing a monochrome photography on the respective colors. In thismethod, different colors of the same channel are not analyzed.Meanwhile, in the hyperspectral development, the focus is shifted togenerate a prism spectrum derived from a magnification chromaticaberration and place spectral information in a moire pattern.

When an image of a white monotone subject is captured with a normalcamera, the captured image is white. If the magnification chromaticaberration occurs, the white color is mixed with a magnificationchromatic aberration of an adjacent pixel, and the color of the imagestill remains white. For that reason, nothing is observed from the whiteimage.

Meanwhile, in the case where a difference between two images, that is,between the first focus image and the second focus image or between theformed image and the defocused image, is observed, a trace of amagnification chromatic aberration can be slightly observed asaberration information in the moire pattern of the interference imagedue to a subtle change in spectral characteristics on a light-reflectingsurface of the subject. This is because when the focus moves, the degreeof the magnification chromatic aberration changes. This aberrationinformation is emphasized by the learning-type pattern conversioncircuit that performs learning so as to emphasize contrast, so that thesubtle change on the surface of the subject, which has been difficult tocapture with the camera in related art, can be captured (see FIG. 16).

[Specific Example of Hyperspectral Development]

Next, a specific example of the hyperspectral development will bedescribed. FIG. 17 is a diagram showing a specific example of thehyperspectral development.

An image A is the first focus image before the hyperspectraldevelopment. An image B is the interference image. In this interferenceimage, a moire pattern can be observed due to a slight diffenrece inspectral characteristics when the focus is shifted. An image C is adeveloped image that has been subjected to a two-dimensionalhyperspectral development. In the developed image, a slight contrast ofthe moire pattern is emphasized, and a state of a change with time thatoccurs around a crack appearing in the image can be observed in detail.This state of the change is difficult to observe in an image capturedwith a normal camera.

Hereinabove, the second embodiment has been described.

Third Embodiment

In the third embodiment, as mentioned above, a stereoscopic developmenttechnique using Δφ will be described.

[Overall Configuration]

First, the overall configuration of an imaging apparatus and adevelopment apparatus according to an embodiment of the presentdisclosure will be described. FIG. 18 is a configuration diagram showingthe overall configuration of an imaging apparatus 100 and a developmentapparatus 204 according to the embodiment of the present disclosure.

The configuration of the imaging apparatus 100 to be used in the thirdembodiment is the same as that of the first or second embodiment, andthus description thereof will be omitted.

The main difference on the development apparatus from the secondembodiment is that the development apparatus 202 is replaced with thedevelopment apparatus 204, in which the de-blurring processing unit 14Bis removed and the diffusion processing unit 11B, the differencecalculation unit 13B, and the component synthesis processing unit 15 areprovided with different functions to be changed into a diffusionprocessing unit 11C, a difference calculation unit 13C, and a componentsynthesis processing unit 15B, respectively. In the followingdescription, constituent elements having the same functions as those ofthe embodiments described above are denoted by the same referencesymbols and description thereof will be omitted.

The development apparatus 204 includes a reception unit 9, anangle-of-view adjustment unit 10, the diffusion processing unit 11C, thedifference calculation unit 13C, and the component synthesis processingunit 15B.

The diffusion processing unit 11C uses a point spread function to changethe first focus image into a digital defocused image in a statedefocused by computing. However, unlike the embodiments described above,the diffusion processing unit 11C uses two point spread functions thatare biased in a right direction and a left direction to be linearlysymmetrical with each other to perform diffusion on the first focusimage. Thus, two digital defocused images for a right image and a leftimage are generated.

In the above description, the diffusion processing unit 11C generatesthe two digital defocused images for the right image and the left image,but the images to be generated are not limited to the right image andthe left image and may be an upper image and a lower image. When thedigital defocused images for the upper image and the lower image aregenerated, a stereoscopic image eventually developed can be viewedstereoscopically in the state of being rotated by 90 degrees.

The difference calculation unit 13C calculates a difference value foreach pixel between each of the two digital defocused images for theright image and the left image, which are generated by the diffusionprocessing unit 11C, and the second focus image, which is supplied fromthe angle-of-view adjustment unit 10, to generate two pieces of maskinformation for the right image and the left image.

Based on the two pieces of mask information for the right image and theleft image, which are generated by the difference calculation unit 13C,and the formed image supplied from the reception unit 9, the componentsynthesis processing unit 15B synthesizes the two developed images ofthe right and left images for stereoscopic viewing.

Hereinabove, the overall configuration has been described.

[Modified Example of Overall Configuration]

Next, an imaging apparatus 103 and a development apparatus 205 that haveconfigurations different from the above configurations will bedescribed. FIG. 19 is a configuration diagram showing the overallconfiguration of the imaging apparatus 103 and the development apparatus205 according to the embodiment of the present disclosure.

The main difference from the configurations described above is that theangle-of-view adjustment unit 10, the diffusion processing unit 11C, andthe difference calculation unit 13C of the development apparatus 204 aremoved to the imaging apparatus 103 side. It should be noted thatdescription on constituent elements having the same functions as thoseof the above constituent elements will be omitted.

The imaging apparatus 103 includes an objective lens 1, a half mirror 2,an imaging device 3, a first-focus-image-capturing controller 4, anoptical path length change filter 5, an imaging device 6, asecond-focus-image-capturing controller 7, the angle-of-view adjustmentunit 10, the diffusion processing unit 11C, the difference calculationunit 13C, and a transmission unit 8C.

The transmission unit 8C transmits the two pieces of mask informationfor the right image and the left image, which are generated by thedifference calculation unit 13C, and the first focus image supplied fromthe first-focus-image-capturing controller 4, to the developmentapparatus 205.

The development apparatus 205 includes a reception unit 9C and acomponent synthesis processing unit 15B.

The reception unit 9C receives the two pieces of mask information forthe right image and the left image and the first focus image, which aretransmitted from the transmission unit 8C of the imaging apparatus 103.The received mask information and first focus image are supplied to thecomponent synthesis processing unit 15B.

Hereinabove, the modified example of the overall configuration has beendescribed.

[Processing Flow]

Subsequently, a processing flow of the stereoscopic development will bedescribed. FIG. 20 is a diagram showing a processing flow of thestereoscopic development.

Here, a processing flow after a time point at which the processing ofthe angle-of-view adjustment unit 10 is terminated will be described.Further, in the processing of generating a stereoscopic image, ageneration process of a left-eye image and a generation process of aright-eye image are not different from each other except that theprocessing for the left and the right are inversed. So, only theprocessing of the left-eye image will be described below. It should benoted that a left point spread function and a right point spreadfunction are linearly symmetrical with each other.

First, the diffusion processing unit 11C uses the left point spreadfunction, a distribution of which is biased to the left side, for thefirst focus image to generate a left digital defocused image.

Subsequently, the difference calculation unit 13C acquires a differencevalue for each pixel between the left digital defocused image and thesecond focus image to generate right-signal mask information.

Next, the component synthesis processing unit 15B synthesizes the firstfocus image and the right-signal mask information to develop a left-eyeimage.

This is the processing flow of the stereoscopic development.

[Stereoscopic Development]

The defocused image contains information on light coming from aplurality of directions as phase information. By using the formed imageto mask light coming from a direction unnecessary for stereoscopicviewing, the light used for the stereoscopic viewing can be separated.This separation can provide a stereoscopic image vertically andhorizontally having an optional angle of convergence (stereoscopiceffect) (see FIG. 21).

However, the angle of convergence actually obtained depends on a focallength or an aperture of the objective lens 1, and the stereoscopiceffect obtained by the stereoscopic development technique according tothe embodiment of the present disclosure is almost equal to thatobtained by a single-lens beam-splitter stereo camera.

Hereinabove, the third embodiment has been described.

Fourth Embodiment

In the fourth embodiment, as mentioned above, a refocusing developmenttechnique using Δφ will be described.

[Overall Configuration]

First, the overall configuration of an imaging apparatus and adevelopment apparatus according to an embodiment of the presentdisclosure will be described. FIG. 22 is a configuration diagram showingthe overall configuration of an imaging apparatus 100 and a developmentapparatus 206 according to the embodiment of the present disclosure.

The configuration of the imaging apparatus 100 to be used in the fourthembodiment is the same as that of each embodiment described above, andthus description thereof will be omitted.

The main difference on the development apparatus 206 from theembodiments described above is that the diffusion processing unit 11 andthe upsampling unit 12 are removed, a blurring and de-blurringprocessing unit 14C and a component synthesis processing unit 15C areprovided with functions described later, and an input unit 18 isadditionally provided. In the following description, constituentelements having the same functions as those of the embodiments describedabove are denoted by the same reference symbols and description thereofwill be omitted.

The development apparatus 206 includes a reception unit 9, anangle-of-view adjustment unit 10, a difference calculation unit 13, theblurring and de-blurring processing unit 14C, the component synthesisprocessing unit 15C, and the input unit 18.

The input unit 18 receives an input of an in-focus position designatedby a user. For example, when the user wants to develop an image in whichthe background of the image is in focus, “Backward” is input, and whenthe user wants to develop an image in which an object located in themiddle of the image is in focus, “Center” is input. When the user wantsto develop an image in which the foreground of the image is in focus,“Forward” is input, and when the user wants to develop an image in whichall areas are in focus, “Deep focus” is input.

The blurring and de-blurring processing unit 14C estimates an opticalpath of light by which an image is formed on a focus plane, by usinglens characteristics of the objective lens 1 that are learned by thelearning-type pattern conversion circuit in advance and based onluminance information and spectral information of the interference imagesupplied from the difference calculation unit 13 and on a position(in-focus position) to be refocused that is supplied from the input unit18. Subsequently, based on the spectral information, more specifically,the magnification chromatic aberration, the point spread function iscaused to act in an image-diffusing direction or an image-convergingdirection. Thus, difference information between an image of alow-frequency component and a refocused image as a target is generated.The difference information is necessary for the refocusing development.The processing for the refocusing development by the blurring andde-blurring processing unit 14C will be described later.

The component synthesis processing unit 15C uses the differenceinformation supplied from the blurring and de-blurring processing unit14C and any one of the first focus image (first low-frequency component)and the second focus image (second low-frequency component) suppliedfrom the reception unit 9 to synthesize a refocused image. Theprocessing for the refocusing development by the component synthesisprocessing unit 15C will be described later.

Hereinabove, the overall configuration has been described.

[Modified Example of Overall Configuration]

Next, an imaging apparatus 104 and a development apparatus 207 that haveconfigurations different from the above configurations will bedescribed. FIG. 23 is a configuration diagram showing the overallconfiguration of the imaging apparatus 104 and the development apparatus207 according to the embodiment of the present disclosure.

The main difference from the configurations described above is that theangle-of-view adjustment unit 10 and the difference calculation unit 13of the development apparatus 206 are moved to the imaging apparatus 104side. It should be noted that description of constituent elements havingthe same functions as those of the above constituent elements will beomitted.

The imaging apparatus 104 includes an objective lens 1, a half mirror 2,an imaging device 3, a first-focus-image-capturing controller 4, anoptical path length change filter 5, an imaging device 6, asecond-focus-image-capturing controller 7, the angle-of-view adjustmentunit 10, the difference calculation unit 13, and a transmission unit 8D.

The transmission unit 8D transmits the interference image generated bythe difference calculation unit 13, the first focus image supplied fromthe first-focus-image-capturing controller 4, and the second focus imagesupplied from the second-focus-image-capturing controller 7 to thedevelopment apparatus 207.

The development apparatus 207 includes a reception unit 9D, a blurringand de-blurring processing unit 14C, a component synthesis processingunit 15C, and an input unit 18.

The reception unit 9D receives the interference image, the first focusimage, and the second focus image that are transmitted from thetransmission unit 8D of the imaging apparatus 104. The receivedinterference image is supplied to the blurring and de-blurringprocessing unit 14C, and the first focus image and the second focusimage are supplied to the component synthesis processing unit 15C.

Hereinabove, the modified example of the overall configuration has beendescribed.

[Principles of Refocusing Development]

Subsequently, the principles of the refocusing development will bedescribed. In the refocusing development, an optical path on which thecaptured image traces after transmitting the objective lens 1 isinversely calculated based on the interference image obtained from thetwo images with different focuses. Thus, an image can be developed at anoptional focus position. The inventor verified by experiments thatimages equal to those captured with lenses of different characteristicscan be developed in a pseudo manner by the refocusing development of,for example, performing refocusing after images are captured orgenerating a deep-focus image being in focus on the entire image.

FIG. 24 is a diagram for describing the principles of the refocusingdevelopment. A first focus image A includes an image IMG1A and an imageIMG2A of a subject. Further, a second focus image B includes an imageIMG1B and an image IMG2B of a subject.

For example, since the image IMG1A and the image IMG1B are obtained byimaging the same subject in different focuses, when their outlines areconnected, an optical path 1 can be estimated.

Subsequently, a refocusing plane C at a focal position where refocusingis intended to be performed is specified, an image IMG1C afterrefocusing can be calculated as a plane on which the optical path 1 andthe refocusing plane C intersect. In the same manner, an optical path 2can also be estimated from the image IMG2A and the image IMG2B, and thusan image IMG2C on the refocusing plane C can be calculated.

Hereinabove, the principles of the refocusing development have beendescribed.

(Method of Estimating Optical Path Using Spectral Information ofInterference Image)

Next, a method of estimating the optical path based on the interferenceimage in the blurring and de-blurring processing unit 14C will bedescribed. Here, the fact that the spectral information that is alsocontained in the interference image has to be used to estimate theoptical path in addition to the luminance information contained in theinterference image will be described.

First, the fact that the optical path is not estimated by only theluminance information contained in the interference image will bedescribed. FIG. 25 is a diagram showing a state where an optical path isnot estimated by using only the luminance information.

FIG. 25 shows the image IMG1A in the first focus image A and the imageIMG1B in the second focus image B, as in FIG. 24 used for describing thesection of the “Principles of Refocusing development”. The size of theimage IMG1A and that of the IMG1B can be calculated from the luminanceinformation of the interference image. However, it is difficult todetermine, from the size of those two images, whether the optical pathis an optical path 1X having an in-focus plane between the first focusand the second focus as shown in the upper part of FIG. 25 or an opticalpath 1Y having no in-focus plane between the first focus and the secondfocus as shown in the lower part of FIG. 25. For that reason, as shownin the upper and lower parts of FIG. 25, in the optical path 1X and theoptical path 1Y, the size of the image IMG1C obtained on the refocusingplane C is not uniquely determined.

Subsequently, the fact that the optical path can be estimated by alsousing the spectral information contained in the interference image willbe described. FIG. 26 is a diagram showing a state where the opticalpath can be estimated by using the luminance information and thespectral information.

FIG. 26 shows the image IMG1A in the first focus image A, the imageIMG1B in the second focus image B, the optical path 1X, and the opticalpath 1Y as shown also in FIG. 25. The difference from FIG. 25 is that aposition through which a light beam passes differs depending onwavelengths due to the magnification chromatic aberration in thevicinity of the outlines of the images IMG1A and IMG1B. For easyunderstanding, FIG. 26 shows red light, green light, and blue light(denoted by R, G, and B, respectively, in FIG. 26) as an example.

Around the image IMG1A, the colors near the outline of the image IMG1Aare arranged in the order of R, G, and B from the inner side. Incontrast to this, around the image IMG1B, the colors near the outline ofthe image IMG1B are arranged in the order of B, G, and R from the innerside in the optical path 1X having an in-focus plane between the firstfocus and the second focus as shown in the upper part of FIG. 26, butconversely arranged in the order of R, G, and B from the inner side inthe optical path 1Y as shown in the lower part of FIG. 26.

When color bleeding in the outline of the image, which occurs by themagnification chromatic aberration, is observed in such a manner, it ispossible to estimate that the optical path is the optical path 1X or theoptical path 1Y.

Hereinabove, the method of estimating the optical path using thespectral information of the interference image has been described.

[Actual Example of In-Focus Position and Magnification ChromaticAberration]

Next, a relationship between the in-focus position and the magnificationchromatic aberration to be generated, when an image is captured, will bedescribed using an actual example. FIG. 27 is a diagram showing a statewhere a condition of the magnification chromatic aberration differs dueto the in-focus position moving backward and forward.

An image A is an image in which not a doll appearing at the center butthe background is in focus. An image B is an image in which the dollappearing at the center is in focus. An image C is an image in which notthe doll appearing at the center but the front side is in focus.Further, each of images A1, B1, C1 is obtained by enlarging the part ofan eye of the doll appearing at the center of each of the images A, B,and C, respectively. In the image B1, the doll is in focus and the colorbleeding due to the magnification chromatic aberration is not recognizedat the eye part. In contrast to this, in the image A1, the backward ofthe doll is in focus, and in the eye part, particularly in the outlineof the upper eyelid, color bleeding of the blue color occurs.Additionally, in the image C1, the forward of the doll is in focus, andin the eye part, particularly in the outline of the upper eyelid, colorbleeding of the yellow color occurs.

As described above, the image A1 and the image C1 have the same degreeof defocusing from the image B1, but have different color bleeding dueto the magnification chromatic aberration. For that reason, it ispossible to determine, based on the different color bleeding, whetherthe in-focus position is located forward or backward of the image evenwhen the image A1 and the image C1 have the same degree of blurring.

Hereinabove, the actual example of the relationship between the in-focusposition and the magnification chromatic aberration to be generated,when an image is captured, has been described.

[Examples of Spectral Information Contained in Interference Image andUse of Point Spread Function]

Next, examples of the spectral information contained in the interferenceimage are shown, and how the point spread function is caused to act (inan image-diffusing direction or an image-converging direction) is shownin each example. FIG. 28 is a diagram showing examples of the spectralinformation in the interference image and showing how the point spreadfunction is caused to act in a diffusing or converging direction foreach example.

In FIG. 28, color bleeding in blue is recognized due to themagnification chromatic aberration in an image appearing in an area A1.For the image in which the color bleeding in blue is observed, the pointspread function is caused to act in the image-diffusing direction, thatis, in a defocus direction.

Further, color bleeding in yellow is recognized due to the magnificationchromatic aberration in an image appearing in an area A2. For the imagein which the color bleeding in yellow is observed, the point spreadfunction is caused to act in the image-converging direction, that is, inan in-focus direction.

In such a manner, in the interference image, the difference in color dueto the magnification chromatic aberration can be determined, and wheninformation on such a difference is learned by the learning-type patternconversion circuit of the blurring and de-blurring processing unit 14C,it is possible to determine whether the image in the interference imageis developed in the diffusing direction or the converging direction.

With this determination, an image that is refocused to an optional focusposition can be developed based on the two images with differentfocuses.

Hereinabove, the examples of the spectral information contained in theinterference image and the use of the point spread function have beendescribed.

[Actual Examples of Refocusing Development]

Next, actual examples of the refocusing development will be described.FIG. 29 is a diagram showing an example of an image in which not a dollbut a background is in focus. This image is the first image to be usedin the refocusing development and corresponds to the first focus image.In this image, not the doll appearing at the center but the backgroundis in focus.

FIG. 30 is a diagram showing an example of an image in which not thedoll but a front side is in focus. This image is the second image to beused in the refocusing development and corresponds to the second focusimage. In this image, not the doll appearing at the center but theforward is in focus and the entire image is defocused.

FIG. 31 is a diagram showing an example of a refocused image obtained byperforming the refocusing development on the images shown in FIGS. 29and 30 to develop an image with the doll being in focus. In thisexample, an image in which the background is blurred and the doll at thecenter is in focus can be developed.

FIG. 32 is a diagram showing an example of a deep-focus image obtainedby performing the refocusing development on the images shown in FIGS. 29and 30 to develop an image with the doll and the background being infocus. Using the refocusing development in such a manner, a deep-focusimage in which subjects at a plurality of different distances arefocused can also be developed.

Hereinabove, the actual examples of the refocusing development have beendescribed.

Fifth Embodiment

Next, an embodiment in which the high resolution development isperformed without the defocus mechanism will be described. In thisembodiment, magnification chromatic aberration correction processing isused instead of the defocus processing. A system for performing themagnification chromatic aberration correction processing is already putto practical use and installed in products, so the high resolutiondevelopment can be implemented by only slightly correcting existingproducts.

[Overall Configuration]

First, the overall configuration of an imaging apparatus and adevelopment apparatus according to an embodiment of the presentdisclosure will be described. FIG. 33 is a configuration diagram showingthe overall configuration of an imaging apparatus 105 and a developmentapparatus 208 according to the embodiment of the present disclosure. Itshould be noted that constituent elements having the same functions asthose of the above constituent elements are denoted by the samereference symbols and description thereof will be omitted.

The imaging apparatus 105 includes an objective lens 1, an imagingdevice 3, a first-focus-image-capturing controller (imaging unit) 4, anaberration correction unit (correction unit) 16, a differencecalculation unit 13D, a high-frequency component extraction unit 17, anda transmission unit 8E.

The aberration correction unit 16 acquires lens information on focus,zoom, and the like of the objective lens 1 and corrects a magnificationchromatic aberration of a formed image based on the acquired lensinformation by a generally-known magnification chromatic aberrationcorrection method.

Specifically, a shift between an R channel and a G channel and a shiftbetween a B channel and the G channel are corrected by digitalprocessing. This step of correcting the shift produces substantially thesame effects as the defocus step, and thus the correction of themagnification chromatic aberration allows the high resolutiondevelopment to be performed without performing the defocus processing.It should be noted that the magnification chromatic aberration isdescribed here, but the present disclosure is not limited thereto and aspherical aberration, a coma aberration, and the like may be corrected.Further, the image may be subjected to a digital optical correction suchas digital de-blurring processing. The image generated here is an imageon which at least one of the aberration correction and the digitaloptical correction is performed.

It should be noted that an image on which the magnification chromaticaberration correction is performed is hereinafter referred to as acorrected image. The corrected image is output to the differencecalculation unit 13D so as to generate an interference image, output tothe high-frequency component extraction unit 17 so as to extract ahigh-frequency component, and output to the transmission unit 8E so asto be used in component synthesis processing in the developmentapparatus 208.

The difference calculation unit 13D calculates a difference for eachpixel between the input first focus image and corrected image, generatesan interference image, and outputs the interference image to thetransmission unit 8E.

The high-frequency component extraction unit 17 extracts only ahigh-frequency component from the corrected image by Wavelet transformor the like. The extraction method is not limited to the Wavelettransform and may be any method as long as the high-frequency componentcan be extracted. The extracted high-frequency component of thecorrected image is transmitted to the transmission unit 8E so as to beused in a de-blurring processing unit 14D of the development apparatus208.

The transmission unit 8E transmits, to the development apparatus 208,coordinate information included in the first focus image, theinterference image, the high-frequency component of the corrected image,and the corrected image. The coordinate information described here isobtained by replacing X-Y coordinate information of the first focusimage with a polar coordinate system (r−θ) with the center image of thefirst focus image being as the center. The polar coordinate system isused because the magnification chromatic aberration is concentricallygenerated based on the center of the image. This information is used forconverging the learning in a learning-type pattern conversion circuit ofthe de-blurring processing unit 14D.

The development apparatus 208 includes a reception unit 9E, thede-blurring processing unit 14D, and a component synthesis processingunit 15.

The reception unit 9E receives the coordinate information, theinterference image, the high-frequency component of the corrected image,and the corrected image from the transmission unit 8E of the imagingapparatus 105, and outputs the coordinate information, the interferenceimage, and the high-frequency component of the corrected image to thede-blurring processing unit 14D and outputs the corrected image to thecomponent synthesis processing unit 15.

The de-blurring processing unit 14D receives inputs of the coordinateinformation, the interference image, and the high-frequency component ofthe corrected image to perform the learning in the learning-type patternconversion circuit so as to inversely calculate the high-frequencycomponent of an image to be developed. For example, in the case wherethe first focus image is an HD size image, the high-frequency componentof a 4K size image is generated by the de-blurring processing. In thecase where the first focus image is a 4K size image, the high-frequencycomponent of an 8K size image is generated. The generated high-frequencycomponent of the developed image is output to the component synthesisprocessing unit 15. Aberration difference information of themagnification chromatic aberration contained in the interference imageand the high-frequency component of the corrected image are input to thelearning-type pattern conversion circuit on a pixel-by-pixel basis toperform processing.

Hereinabove, the overall configuration has been described. It should benoted that the aberration correction unit 16, the difference calculationunit 13D, and the high-frequency component extraction unit 17 areincluded in the imaging apparatus 105 in the above configuration.However, all of those components may be included in the developmentapparatus 208, the aberration correction unit 16 may be included in theimaging apparatus 105, and the aberration correction unit 16 and thedifference calculation unit 13D may be included in the imaging apparatus105.

Specific Examples

Next, an example in which the high resolution development is performedusing the high resolution development technique according to theembodiment of the present disclosure will be described.

FIG. 34 is a diagram showing that a 4K image A as a reference is used togenerate a first focus image B, and the high resolution development isperformed on the first focus image B to generate a developed image C.Comparing the reference 4K image A with the developed image C, it isfound that there is not a large difference therebetween.

Sixth Embodiment

In the first embodiment and the fifth embodiment described above, theholographic development to achieve high resolution in a spatialdirection by using the interference image generated from the first focusimage (formed image) and the second focus image (defocused image) hasbeen described.

In contrast to this, in this embodiment, a high frame rate developmenttechnique in which the theory of the holographic development isbroadened in a time direction to increase a frame rate of a capturedmoving image will be described.

In the high frame rate development, as shown in FIG. 35, an imagecaptured at a shutter speed of 1/60N seconds is used. The shutter speedof 1/60N seconds is based on a normal shutter speed of 1/60 seconds andis 1/N of that shutter speed (N is positive number). This image isreferred to as a 1/N shutter image.

Additionally, an image captured at a shutter speed of (N−1)/60 N secondsis also used. The shutter speed of (N−1)/60 N seconds is the remainingtime obtained by subtracting the 1/60N seconds from the 1/60 seconds.This image is referred to as an (N−1)/N shutter image. Those two imagesare subjected to the high frame rate development to develop a high framerate video equivalent to a video captured at an N×-speed, that is, at ahigh shutter speed of 1/60N seconds.

This technique has an advantage that a high frame rate video can begenerated with a camera capable of processing image data of a total oftwo images in 1/60 seconds by shutter control without using a cameraexclusively used for a high frame rate. The total of two images are animage of an HD resolution that is captured at a shutter speed of 1/60Nseconds and an image of an HD resolution that is captured at a shutterspeed of (N−1)/60 N seconds.

In this technique, however, an extremely high frame rate causes a largeincrease in block noise, and thus the frame rate for development islimited. Further, the resolution of a generated video is also affectedby the dynamic range of the imaging device.

[Principles of High Frame Rate Development]

Next, the principles of the high frame rate development will bedescribed. Here, the case where N=3 is described as an example. FIG. 36is a diagram for describing the principles of the high frame ratedevelopment. Here, two types of images captured in 1/60 seconds arereferred to as a ⅓ shutter image and a ⅔ shutter image.

Based on a frame number of an 3×-speed image to be developed, the frameof an image to be captured is hereinafter referred to as follows asappropriate. First, a first frame (Frame #1) of the ⅓ shutter image iscaptured, and second to third frames of the ⅔ shutter image (Frames #2to 3; corresponding to the second and third frames of the 3×-speedimage) are Subsequently captured. Next, a fourth frame (Frame #4) of the⅓ shutter image is captured, and fifth to sixth frames of the ⅔ shutterimage are captured in the stated order.

(Generation of Pseudo-Interpolation Frame)

In the case where the 3×-speed image is developed from the ⅓ shutterimage and the ⅔ shutter image, first, an amount of a movement betweenthe first frame and the fourth frame of the ⅓ shutter image is analyzed.

Subsequently, one image of the first frame and the fourth frame issimply moved in its moving direction based on the amount of the movementbetween those frames by using linear interpolation by pattern matching.By the linear interpolation, pseudo-interpolation frames correspondingto a second frame (Frame #2) and a third frame (Frame #3) of the3×-speed image are generated.

(Calculation of Interpolation Error)

Since the interpolation is performed by a simple linear movement in thetwo interpolation frames, a rotational movement or an acceleratingmovement of the subject, a movement in a depth direction of the screen,and the like are not precisely interpolated. For that reason, in theimage of the interpolated frame, many interpolation errors occur in anedge portion.

If those two pseudo-interpolation frames can be ideally interpolated, acombined value of those two interpolation frames has to be equal to thesecond to third frames of the ⅔ shutter image.

However, the interpolation is not ideally performed, and thus adifference between the combined value of those two interpolation framesand images of the second to third frames of the ⅔ shutter image isgenerated. In this regard, this difference is used to acquire acumulative total value of the errors corresponding to the twopseudo-interpolation frames. The cumulative total value of the errors isto be an interference image.

(Learning in Learning-Type Pattern Conversion Circuit)

The second frame and third frame generated as the pseudo-interpolationframes are generated by the linear interpolation by simply moving oneimage of the first and fourth frames in its moving direction based onthe amount of the movement between the frames.

For that reason, those pseudo-second and third frames and the second andthird frames actually captured with a camera of a 3×-speed high framerate have a correlation in error.

Actually, the following tendency is observed: in the same coordinates ofthe corresponding frames, errors with the same tendency occur and alarger movement amount increases errors.

A relationship between the tendency of error generation in thepseudo-interpolation frames and the interference image that is thecumulative total value of the errors is learned in the learning-typepattern conversion circuit, so that the error amount in eachinterpolation frame can be corrected.

(Allocation of Amount of Error Correction)

Allocation of the amount of error correction to the second frame and thethird frame that are the pseudo-interpolation frames can be performedbecause there is a certain tendency in the generation of the errors ofthose two frames.

The allocation of the amount of error correction is feasible because thepseudo-interpolation frames are generated by simply moving in parallel aframe (first frame or fourth frame) of one ⅓ shutter image. If anaverage between the two frames of the first frame and the fourth frameis obtained and interpolation is performed, a certain tendency in thegeneration of the errors does not occur and an appropriate allocation ofthe amount of error correction becomes difficult to perform.

(Notes of Correction)

Notes of the correction in the high frame rate development reside inthat the priority is given not to obtain a high-quality interpolationframe but to perform interpolation so as to obtain errors of the sametendency, in the stage of generating the pseudo-interpolation frames.The error correction for obtaining a high-quality interpolation frame isperformed anew using the interference image at the last of the highframe rate development.

[Configuration of Imaging Apparatus]

Next, the configuration of an imaging apparatus for performing the highframe rate development will be described. FIG. 37 is a functional blockdiagram showing the configuration of an imaging apparatus 500.

The imaging apparatus 500 includes an imaging unit 501, an imagingcontroller 502, a pseudo-interpolation image generation unit 503, anaddition processing unit 504, an interference image generation unit 505,a pattern conversion unit 506, and an error correction unit 507.

The imaging unit 501 captures an image at a shutter speed of 1/60Nseconds and an image at a shutter speed of (N−1)/60 N seconds, per 1/60seconds.

The imaging controller 502 controls the imaging unit 501 to capture animage at a predetermined shutter speed and notifies the patternconversion unit 506 of a frame number of a captured image (frame). Thisis because the pattern conversion unit 506 allocates different errorsfor each frame and thus needs the frame number.

The pseudo-interpolation image generation unit 503 receives inputs oftwo images captured at a 1/N shutter image, that is, an image capturedat time t and an image captured at time t− 1/60 (for example, the firstframe and the fourth frame when N=3), and performs a movement analysisusing those two images. It should be noted that the imaging apparatus500 includes a frame memory for storing the image captured at time t andthe image captured at time t− 1/60.

The pseudo-interpolation image generation unit 503 performs linearinterpolation to generate (N−1) pieces of pseudo-interpolation images.The pseudo-interpolation image generation unit 503 supplies thegenerated pseudo-interpolation images to the addition processing unit504 and the error correction unit 507.

The addition processing unit 504 adds the (N−1) pieces ofpseudo-interpolation images supplied from the pseudo-interpolation imagegeneration unit 503 to be integrated into one addition image.Subsequently, the addition processing unit 504 supplies the integratedaddition image to the interference image generation unit 505.

The interference image generation unit 505 receives inputs of theaddition image and the (N−1)/N shutter image to calculate a differencebetween those two images. The difference is supplied to the patternconversion unit 506 as an interference image (error cumulative value).

The pattern conversion unit 506 includes a learning-type patternconversion circuit. In the learning-type pattern conversion circuit, asdescribed above, a relationship between the tendency of error generationin the pseudo-interpolation frames and the interference image as theerror cumulative value is obtained in advance through learning. Thepattern conversion unit 506 generates an error correction amount of eachframe by using the interference image supplied from the interferenceimage generation unit 505 and the frame number supplied from the imagingcontroller 502. The pattern conversion unit 506 supplies the generatederror correction amount to the error correction unit 507.

The error correction unit 507 applies the error correction amountsupplied from the pattern conversion unit 506 to thepseudo-interpolation images supplied from the pseudo-interpolation imagegeneration unit 503 to correct the errors, and generates (N−1) pieces offrames that form an N×-speed image. The error correction unit 507 formsperfect N pieces of frames together with the 1/N shutter image acquiredfrom the imaging unit 501 and outputs the image.

Hereinabove, the configuration of the imaging apparatus 500 has beendescribed.

[Another Configuration of Present Disclosure]

It should be noted that the present disclosure can have the followingconfigurations.

(1) An imaging apparatus, including:

an imaging unit configured to capture two images that are different fromeach other by a predetermined amount of an optical distance (focus)between an objective lens and an imaging device having a firstresolution; and

a transmission unit configured to transmit the captured images.

(2) The imaging apparatus according to (1), in which

the imaging unit includes

-   -   a formed-image-capturing device configured to capture a formed        image on which light passing through the objective lens is        focused, and    -   a defocused-image-capturing device configured to capture a        defocused image on which the light is defocused based on the        predetermined amount.        (3) The imaging apparatus according to (1), further including:

an angle-of-view adjustment unit configured to equalize angles of viewof the two images, one of the two images being a first focus image andthe other image being a second focus image;

a diffusion unit configured to generate, as a digital defocused image,an image obtained by diffusing and enlarging the first focus image tohave a second resolution higher than the first resolution;

an upsampling unit configured to generate, as an enlarged defocusedimage, an image obtained by upsampling the second focus image to havethe second resolution; and

a difference calculation unit configured to generate, as an interferenceimage, a difference for each pixel between the digital defocused imageand the enlarged defocused image, in which

the transmission unit is configured to transmit the first focus imageand the interference image.

(4) The imaging apparatus according to (1), further including:

an angle-of-view adjustment unit configured to equalize angles of viewof the two images, one of the two images being a first focus image andthe other image being a second focus image;

a diffusion unit configured to generate, as a digital defocused image,an image obtained by diffusing the first focus image; and

a difference calculation unit configured to generate, as an interferenceimage, a difference for each pixel between the digital defocused imageand the second focus image, in which

the transmission unit is configured to transmit the first focus imageand the interference image.

(5) The imaging apparatus according to (3) or (4), in which

the diffusion unit is configured to diffuse the formed image by a pointspread function.

(6) The imaging apparatus according to (1), further including:

an angle-of-view adjustment unit configured to equalize angles of viewof the two images, one of the two images being a first focus image andthe other image being a second focus image;

a diffusion unit configured to generate, as a first digital defocusedimage, an image obtained by diffusing the first focus image by a firstfunction and generate, as a second digital defocused image, an imageobtained by diffusing the first focus image by a second function, thefirst function and the second function being linearly symmetrical witheach other; and

a difference calculation unit configured to generate, as first maskinformation, a difference for each pixel between the first digitaldefocused image and the second focus image and generate, as second maskinformation, a difference for each pixel between the second digitaldefocused image and the second focus image, in which

the transmission unit is configured to transmit the first focus image,the first mask information, and the second mask information.

(7) The imaging apparatus according to (1), further including:

an angle-of-view adjustment unit configured to equalize angles of viewof the two images, one of the two images being a first focus image andthe other image being a second focus image; and

a difference calculation unit configured to generate, as an interferenceimage, a difference for each pixel between the first focus image and thesecond focus image, in which

the transmission unit is configured to transmit the first focus image,the second focus image, and the interference image.

(8) The imaging apparatus according to any one of (3) to (7), in which

the first focus image is a formed image that is in focus, and

the second focus image is a defocused image that is defocused by apredetermined amount from the in-focus position.

(9) An imaging apparatus, including:

an imaging unit configured to capture an image;

a correction unit configured to perform at least one of an aberrationcorrection and a digital optical correction on the image to generate acorrected image;

a difference calculation unit configured to generate, as an interferenceimage, a difference for each pixel between the image and the correctedimage; and

a transmission unit configured to transmit coordinate informationcontained in the image, the interference image, and the corrected image.

(10) An imaging method, including:

capturing two images that are different from each other by apredetermined amount of an optical distance (focus) between an objectivelens and an imaging device; and

transmitting the captured images.

[Supplemental Matters]

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing apparatus comprising:circuitry configured to generate a plurality of first interpolationframes using a first frame and a second frame of an input image, whereina first shutter speed of the first frame and a second shutter speed ofthe second frame are the same; generate an additional frame by addingthe plurality of first interpolation frames; generate an interferenceimage from the additional frame and a third frame of the input image,wherein a third shutter speed of the third frame is slower than thefirst shutter speed of the first frame; generate an error correctionamount for each of the plurality of first interpolation frames accordingto the third frame; and generate a plurality of second interpolationframes chronologically positioned between the first frame and the secondframe by applying each of the error correction amount to each of theplurality of first interpolation frames.
 2. The image processingapparatus according to claim 1, wherein the circuitry is furtherconfigured to form an output image constructed with the first frame andthe plurality of second interpolation frames.
 3. The image processingapparatus according to claim 2, wherein a frame rate of the output imageis higher than a frame rate of the input image.
 4. The image processingapparatus according to claim 1, wherein a number of the firstinterpolation frame and a number of the second interpolation frame isthe same.
 5. The image processing apparatus according to claim 1,further comprising an imaging device configured to generate the inputimage.
 6. The image processing apparatus according to claim 1, whereinthe circuitry is further configured to generate the plurality of firstinterpolation frames by analyzing a movement between the first frame andthe second frame.
 7. The image processing apparatus according to claim6, wherein the movement is a simple movement between the first frame andthe second frame.
 8. The image processing apparatus according to claim6, wherein the movement is a linear movement between the first frame andthe second frame.
 9. The image processing apparatus according to claim1, wherein the circuitry is further configured to generate the errorcorrection amount for each of the plurality of first interpolationframes using a learning-type pattern conversion scheme.
 10. A method forprocessing an image, the method comprising: generating a plurality offirst interpolation frames using a first frame and a second frame of aninput image, wherein a first shutter speed of the first frame and asecond shutter speed of the second frame are the same; generating anadditional frame by adding the plurality of first interpolation frames;generating an interference image from the additional frame and a thirdframe of the input image, wherein a third shutter speed of the thirdframe is slower than the first shutter speed of the first frame;generating an error correction amount for each of the plurality of firstinterpolation frames according to the third frame; and generating aplurality of second interpolation frames chronologically positionedbetween the first frame and the second frame by applying each of theerror correction amount to each of the plurality of first interpolationframes.
 11. The method according to claim 10, further comprising:forming an output image constructed with the first frame and theplurality of second interpolation frames.
 12. The method according toclaim 11, wherein a frame rate of the output image is higher than aframe rate of the input image.
 13. The method according to claim 10,wherein a number of the first interpolation frame and a number of thesecond interpolation frame is the same.
 14. The method according toclaim 10, wherein generating the plurality of first interpolation framesincludes analyzing a movement between the first frame and the secondframe.
 15. The method according to claim 14, wherein the movement is asimple movement between the first frame and the second frame.
 16. Themethod according to claim 14, wherein the movement is a linear movementbetween the first frame and the second frame.
 17. The method accordingto claim 10, wherein generating the error correction amount for each ofthe plurality of first interpolation frames includes using alearning-type conversion scheme.
 18. A non-transitory computer readablemedium storing program code for processing an image, the program codebeing executable by a processor to perform operations comprising:generating a plurality of first interpolation frames using a first frameand a second frame of an input image, wherein a first shutter speed ofthe first frame and a second shutter speed of the second frame are thesame; generating an additional frame by adding the plurality of firstinterpolation frames; generating an interference image from theadditional frame and a third frame of the input image, wherein a thirdshutter speed of the third frame is slower than the first shutter speedof the first frame; generating an error correction amount for each ofthe plurality of first interpolation frames according to the thirdframe; and generating a plurality of second interpolation frameschronologically positioned between the first frame and the second frameby applying each of the error correction amount to each of the pluralityof first interpolation frames.